Add autoloading tutorial #3037

shink · 2024-09-05T08:52:57Z

This PR adds a tutorial about the out-of-tree extension autoloading mechanism introduced in pytorch/pytorch#122468 and implemented in pytorch/pytorch#127074

cc: @jgong5 @Yikun @hipudding @FFFrog @albanD

pytorch-bot · 2024-09-05T08:53:00Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/tutorials/3037

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit d45477c with merge base d862a95 ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

svekars · 2024-09-05T15:47:44Z

Please make sure to review the Tutorial Submission Policy

albanD

Thanks for writing a tutorial on this. I think it is indeed a good topic to cover.
I think it would be good to take a step back and clarify who the audience is for this and focus the content on providing the information that audience is looking for.

In particular, my expectation is that the audience are the third party extension writers who should use this feature to improve seamless integration within PyTorch. In that case,
I will keep the intro at the top but make the "How to apply this to out-of-tree extensions the main section after that.

The How it works is covered in the RFC IIRC, maybe linking to it would be enough? I do agree that pointing out the env variable to disable this feature is a good thing to mention here though.

For the examples section, that makes sense to have this at the end as "example projects using this feature as well" but I would limit it to code pointers into how they did the integration on their github repos code. I don't think the code samples add much?

shink · 2024-09-06T01:27:11Z

@svekars @albanD Thanks so much for your comments!

Please make sure to review the Tutorial Submission Policy

Oh I'm so sorry, I will make sure to review this policy and adhere to it.

Thanks for writing a tutorial on this. I think it is indeed a good topic to cover. I think it would be good to take a step back and clarify who the audience is for this and focus the content on providing the information that audience is looking for.

In particular, my expectation is that the audience are the third party extension writers who should use this feature to improve seamless integration within PyTorch. In that case, I will keep the intro at the top but make the "How to apply this to out-of-tree extensions the main section after that.

The How it works is covered in the RFC IIRC, maybe linking to it would be enough? I do agree that pointing out the env variable to disable this feature is a good thing to mention here though.

For the examples section, that makes sense to have this at the end as "example projects using this feature as well" but I would limit it to code pointers into how they did the integration on their github repos code. I don't think the code samples add much?

Thanks for your professional review! I totally agree with your suggestion. I'm making changes.

albanD

Re-shuffle looks good. Thanks for the quick update.
Only the TODO in the examples section but the rest sounds good to me at a high level. Will let @svekars review in case there is any finer grain issue needed.

shink · 2024-09-09T03:03:27Z

@jgong5 Please review the HPU example section, thanks!

advanced_source/python_extension_autoload.rst

jgong5 · 2024-09-09T03:37:19Z

_static/img/python_extension_autoload_impl.png

The diagram seems missing a pointer from import torch to the entry points to load - the loading of entry points is triggered by import torch, right?

yes, I will make some changes to this diagram

@jgong5 updated, please have a look

Co-authored-by: Jiong Gong <[email protected]>

shink · 2024-09-09T03:52:31Z

advanced_source/python_extension_autoload.rst

+`habana_frameworks.torch`_ is a Python package that enables users to run
+PyTorch programs on Intel Gaudi via the PyTorch ``HPU`` device key.
+``import habana_frameworks.torch`` is no longer necessary after this mechanism
+is applied.
+
+.. _habana_frameworks.torch: https://docs.habana.ai/en/latest/PyTorch/Getting_Started_with_PyTorch_and_Gaudi/Getting_Started_with_PyTorch.html
+
+.. code-block:: diff
+
+    import torch
+    import torchvision.models as models
+    - import habana_frameworks.torch # <-- extra import
+    model = models.resnet50().eval().to("hpu")
+    input = torch.rand(128, 3, 224, 224).to("hpu")
+    output = model(input)
+


@jgong5 should the implementation example of this mechanism be added here?

Yes, please leave it here for now. @bsochack is the habana bridge ready for autoloading?

It is implemented and verified. Not yet released.

@bsochack we need an implementation example here, can you provide one? Thanks!

Any suggestion how it should be provided?

The only challenge that we had is the circular dependencies. We solved it in the way below. Other than that, it worked out of box.

habana_frameworks/init.py:

import os is_loaded = False # A member variable of habana_frameworks module to track if our module has been imported def __autoload(): # This is an entrypoint for pytorch autoload mechanism # If the following confition is true, that means our backend has already been loaded, either explicitly # or by the autoload mechanism and importing it again should be skipped to avoid circular imports global is_loaded if is_loaded: return import habana_frameworks.torch

habana_frameworks/torch/init.py:

import os # This is to prevent torch autoload mechanism from causing circular imports import habana_frameworks habana_frameworks.is_loaded = True

and one addition in setup.py:

entry_points={ "torch.backends": [ "device_backend = habana_frameworks:__autoload", ], },

@bsochack thanks so much for your quick reply!

The circular dependency issue is worth noting, I will state this in this doc.

I noticed habana_frameworks is not an open-source project, so are there any public links about the autoloading mechanism? It's OK if not.

Right, habana_frameworks is not open source and there is no official release with the above change.
Btw, it requires PT 2.5 (not yet released) so it is hard to provide a real example.

@bsochack OK it's alright

@bsochack updated, please take a look at the HPU section

Co-authored-by: Svetlana Karslioglu <[email protected]>

shink · 2024-09-14T02:57:52Z

@svekars Thanks so much for your professional and patient review! I made some changes which can be found at: https://github.com/pytorch/tutorials/pull/3037/files/4d44a78d348b918a5f6a3ff60f804b330e983546..e425fe90aeca2b734c390ddd4e0fadc1f0f715fd

shink · 2024-09-18T06:49:22Z

Hi @svekars, just a gentle reminder about this PR. I'm hoping your review and thanks for your time!

svekars · 2024-09-23T18:24:37Z

It's LGTM. One question - it is a prototype feature, correct? Can we put it under the prototype_source, then?

shink · 2024-09-24T01:24:06Z

@svekars Yes, I'm doing this. But will it always be under the prototype_source, or will it be moved back after pytorch2.5 is released?

shink · 2024-09-25T02:42:38Z

@svekars The move is done, could you please review it again?

svekars · 2024-09-25T17:07:01Z

@shink my understanding is that this feature is released as a prototype in 2.5 so it should remain there until Beta status is reached in one of the future releases.

svekars

LGTM!

shink · 2024-09-27T08:33:10Z

@svekars thanks for your approval and can this pr be merged? :D

docs: add autoload

ee33c2d

facebook-github-bot added the cla signed label Sep 5, 2024

shink added 4 commits September 5, 2024 17:07

update

6215e6e

update

9f23f16

update

37c72ee

update

67ad7ff

svekars added the new tutorial label Sep 5, 2024

Merge branch 'main' into docs/autoload

22ddca1

albanD reviewed Sep 5, 2024

View reviewed changes

shink added 2 commits September 6, 2024 09:59

update

891753e

Merge branch 'main' into docs/autoload

54dd911

shink marked this pull request as draft September 6, 2024 02:53

shink added 2 commits September 6, 2024 11:09

update

5c9d78b

update

aa47a21

svekars mentioned this pull request Sep 6, 2024

💡 [REQUEST] - Tutorial on out-of-tree extension autoloading mechanism #3039

Closed

albanD reviewed Sep 6, 2024

View reviewed changes

shink added 5 commits September 9, 2024 10:43

Merge branch 'pytorch:main' into docs/autoload

7caaad8

update

b4e884d

update

523d289

update

b3766ed

update

5d9adfb

jgong5 reviewed Sep 9, 2024

View reviewed changes

shink and others added 3 commits September 9, 2024 11:42

Update advanced_source/python_extension_autoload.rst

5d36b03

Co-authored-by: Jiong Gong <[email protected]>

Update advanced_source/python_extension_autoload.rst

9123d82

Co-authored-by: Jiong Gong <[email protected]>

Update advanced_source/python_extension_autoload.rst

e726565

shink commented Sep 9, 2024

View reviewed changes

shink and others added 15 commits September 14, 2024 09:53

Update advanced_source/python_extension_autoload.rst

a48cfc4

Co-authored-by: Svetlana Karslioglu <[email protected]>

Update advanced_source/python_extension_autoload.rst

d1217dc

Co-authored-by: Svetlana Karslioglu <[email protected]>

Update advanced_source/python_extension_autoload.rst

23cfef4

Co-authored-by: Svetlana Karslioglu <[email protected]>

Update advanced_source/python_extension_autoload.rst

3c0c1e0

Co-authored-by: Svetlana Karslioglu <[email protected]>

Update advanced_source/python_extension_autoload.rst

dda22c4

Co-authored-by: Svetlana Karslioglu <[email protected]>

Update advanced_source/python_extension_autoload.rst

0a47d48

Co-authored-by: Svetlana Karslioglu <[email protected]>

Update advanced_source/python_extension_autoload.rst

bcbe0f6

Co-authored-by: Svetlana Karslioglu <[email protected]>

Update advanced_source/python_extension_autoload.rst

80c8683

Co-authored-by: Svetlana Karslioglu <[email protected]>

Update advanced_source/python_extension_autoload.rst

2ba51d0

Co-authored-by: Svetlana Karslioglu <[email protected]>

Merge branch 'main' into docs/autoload

5ea5a36

Update advanced_source/python_extension_autoload.rst

f8365e8

Co-authored-by: Svetlana Karslioglu <[email protected]>

Update advanced_source/python_extension_autoload.rst

0cc9850

Co-authored-by: Svetlana Karslioglu <[email protected]>

Update advanced_source/python_extension_autoload.rst

33c60cc

Co-authored-by: Svetlana Karslioglu <[email protected]>

Update advanced_source/python_extension_autoload.rst

ee5c353

Co-authored-by: Svetlana Karslioglu <[email protected]>

update

e425fe9

svekars added the 2.5 PR related to version 2.5 label Sep 23, 2024

Merge branch 'main' into docs/autoload

f1018e3

shink added 2 commits September 24, 2024 09:24

Merge branch 'main' into docs/autoload

ebfcbff

move to prototype_source

0b52b02

Merge branch 'main' into docs/autoload

9a1b2f7

svekars approved these changes Sep 25, 2024

View reviewed changes

Merge branch 'main' into docs/autoload

d45477c

svekars merged commit 4b22e03 into pytorch:main Oct 9, 2024
17 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add autoloading tutorial #3037

Add autoloading tutorial #3037

shink commented Sep 5, 2024 •

edited

Loading

pytorch-bot bot commented Sep 5, 2024 •

edited

Loading

svekars commented Sep 5, 2024

albanD left a comment

shink commented Sep 6, 2024

albanD left a comment

shink commented Sep 9, 2024

jgong5 Sep 9, 2024

shink Sep 9, 2024

shink Sep 10, 2024

shink Sep 9, 2024

jgong5 Sep 9, 2024

bsochack Sep 9, 2024

shink Sep 9, 2024

bsochack Sep 9, 2024

shink Sep 9, 2024

bsochack Sep 9, 2024

shink Sep 9, 2024

shink Sep 10, 2024

bsochack Sep 10, 2024

shink commented Sep 14, 2024

shink commented Sep 18, 2024

svekars commented Sep 23, 2024

shink commented Sep 24, 2024

shink commented Sep 25, 2024

svekars commented Sep 25, 2024 •

edited

Loading

svekars left a comment

shink commented Sep 27, 2024 •

edited

Loading

Add autoloading tutorial #3037

Add autoloading tutorial #3037

Conversation

shink commented Sep 5, 2024 • edited Loading

pytorch-bot bot commented Sep 5, 2024 • edited Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/tutorials/3037

✅ No Failures

svekars commented Sep 5, 2024

albanD left a comment

Choose a reason for hiding this comment

shink commented Sep 6, 2024

albanD left a comment

Choose a reason for hiding this comment

shink commented Sep 9, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shink commented Sep 14, 2024

shink commented Sep 18, 2024

svekars commented Sep 23, 2024

shink commented Sep 24, 2024

shink commented Sep 25, 2024

svekars commented Sep 25, 2024 • edited Loading

svekars left a comment

Choose a reason for hiding this comment

shink commented Sep 27, 2024 • edited Loading

shink commented Sep 5, 2024 •

edited

Loading

pytorch-bot bot commented Sep 5, 2024 •

edited

Loading

svekars commented Sep 25, 2024 •

edited

Loading

shink commented Sep 27, 2024 •

edited

Loading